Image sensors are semiconductor devices that capture and process light into electronic signals for forming still images or video. Their use has become prevalent in a variety of consumer, industrial, and scientific applications, including digital cameras and camcorders, hand-held mobile devices, webcams, medical applications, automotive applications, games and toys, security and surveillance, pattern recognition, and automated inspection, among others. The technology used to manufacture image sensors has continued to advance at a rapid pace.
There are two main types of image sensors available today: Charge-Coupled Device (“CCD”) sensors and Complementary Metal Oxide Semiconductor (“CMOS”) sensors. In either type of image sensor, a light gathering photosite is formed on a semiconductor substrate and arranged in a two-dimensional array. The photosites, generally referred to as picture elements or “pixels,” convert the incoming light into an electrical charge. The number, size, and spacing of the pixels determine the resolution of the images generated by the sensor.
Modern image sensors typically contain millions of pixels in the pixel array to provide high-resolution images. The image information captured in each pixel, e.g., raw pixel data in the Red, Green, and Blue (“RGB”) color space, is transmitted to an Image Signal Processor (“ISP”) or other Digital Signal Processor (“DSP”) where it is processed to generate a digital image.
The quality of the digital images generated by an image sensor depends mostly on its sensitivity and a host of other factors, such as lens-related factors (flare, chromatic aberration), signal processing factors, time and motion factors, semiconductor-related factors (dark currents, blooming, and pixel defects), and system control-related factors (focusing and exposure error, white balance error). White balance error, for example, causes poor color reproduction and can easily deteriorate image quality if not corrected for.
White balance in an image sensor device refers to the adjustment of the primary colors, e.g., Red, Green, and Blue, in images captured by the device so that a captured image that appears white for the device also appears white for the Human Visual System (“HVS”). The discrepancy in colors perceived by an image sensor device and the HVS arises out of the many light sources available and their different color temperatures. While the HVS is proficient in adapting to different light sources illuminating a scene, commonly referred to as the scene illuminants, image sensors are not capable of accurately capturing color in all color temperatures. For example, a white paper may be captured by an image sensor as slightly reddish under a household light bulb or as bluish under daylight. The same white paper is perceived as white by the HVS under different scene illuminants.
To emulate the HVS, white balance must be performed in image sensor devices. In addition, image sensor devices must also perform color correction in order to improve the accuracy of color reproduction. Color correction is required because the spectral sensitivity of image sensors differs from the color matching functions of the HVS. The RGB values generated by image sensor devices are also device-dependent, i.e., different devices produce different RGB responses for the same scene.
In order to preserve color fidelity or teach an image sensor device how to see as the HVS expects colors to look, color correction is performed to establish the relationship between device-dependent RGB values and device-independent values. The device-independent values are calculated on the “CIEXYZ” color space, which is based on the International Commission on Illumination (“CIE”) standard observer color-matching functions.
For white balance and color correction to be performed accurately on image sensor devices, the scene illuminant must be known. In general, there are two ways to obtain the scene illuminant information: measuring the color of the scene illuminant, and estimating it from captured images. Most practically, the scene illuminant information is estimated using various approaches.
For example, earlier work for estimating scene illuminants proposed that the white point of an image corresponds to the average image chromaticity. Another earlier proposal was to take the maximum pixel responses in the Red, Green, and Blue color channels individually and use them as the white point estimate. Both proposals suffer from poor estimation performance.
More recent approaches for estimating scene illuminants rely on the assumption that the range of colors measured by an image sensor device (or the eye) depends on the color of the illuminant, e.g., the reddest red color cannot occur under the bluest light. These approaches evaluate the following question: “How likely is it that a given illuminant would have produced the colors of a given image generated by the image sensor device?” In one approach, referred to as “Color in Perspective,” this question is answered by developing models of likely surface colors and illuminant colors and solving a constraint satisfaction problem. This approach is computationally intensive, time consuming, and sensitive to color inconsistencies in captured images.
In another approach, referred to as “Color in Correlation” and described in U.S. Pat. No. 6,038,339, the idea is to exploit the correlation that exists between image colors and scene illuminants, e.g., the reddest image sensor device measurements strongly correlate with the reddest light. In this approach, a correlation matrix memory is built to correlate the data from any image, e.g., an RGB image captured by an image sensor device, to the set of possible scene illuminants. This requires a priori sets of training data corresponding to different known illuminants. Each illuminant is associated with a reference gamut generated by RGB values from the image sensor device in training. Data from a given image captured by the image sensor device are then compared to each reference gamut. The chosen (i.e., estimated) scene illuminant is the one that maximizes the correlation between the image data and the training data.
This Color in Correlation approach, however, is also computationally intensive, time consuming, and requires significant storage. The illuminant correlation matrix needs to be stored in memory and multiplied by a chromaticity vector for each captured image. Depending on the number of illuminants that are used to build the correlation matrix, this could add significant storage and computational costs to image sensor devices. With device manufacturers pushing for lower costs and higher quality, there is a need to provide as accurate white balance and color correction as possible without draining the device resources.
Accordingly, it would be desirable to provide an apparatus and method for estimating scene illuminants that are capable of achieving high white balance and color correction performance with low storage and computational requirements.